Learning Multi-Modal Nonlinear Embeddings: Performance Bounds and an Algorithm

While many approaches exist in the literature to learn low-dimensional representations for data collections multiple modalities, generalizability of multi-modal nonlinear embeddings previously unseen is a rather overlooked subject. In this work, we first present theoretical analysis learning supervised setting. Our performance bounds indicate that successful generalization classification and retrieval problems, regularity interpolation functions extending embedding whole space as important between-class separation cross-modal alignment criteria. We then propose representation algorithm motivated by these findings, where training samples are optimized jointly with Lipschitz interpolators. Experimental comparison recent single-modal algorithms suggests proposed method yields promising image image-text applications.